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UPGRADING AMERICA'S USE OF INFORMATION 
TO IMPROVE STUDENT PERFORMANCE 



Margaret Heritage, John Lee, Eva Chen, 
and Debbie LaTorre 

University of California, Los Angeles 

Policymakers and the public demand high levels of performance for all 
students. The No Child Left Behind (NCLB) legislation requires that all children 
meet specific state-established standards, and includes a range of consequences for 
those schools where students fail to meet the requirements. As expectations rise for 
student performance, the issue of how to improve student achievement becomes a 
critical consideration. 

To ensure that all groups of students are making progress and meeting 
standards, NCLB mandates annual reporting of the performance of sub groups of 
students on large-scale achievement tests. However, reporting assessment data in 
itself will not necessarily lead to effective actions by educators to improve student 
achievement. A critical determinant of how well school personnel use assessment 
results to improve student achievement is their capacity to store, retrieve, integrate 
and analyze assessment and other data. 

School data often exists in disparate forms and locations, making it difficult to 
organize efficiently and to retrieve quickly (Thorn, 2001). Many computer systems 
are outdated and inadequate, and schools and districts often lack appropriate user- 
friendly software for data analysis (Bernhardt 2004; Wayman, Stringfield, & 
Yakimowski, 2004). Furthermore, educators frequently lack the skills of to make 
effective use of data (Baker, 2003; Choppin, 2002; Cizek, 2000; Cromey, 2000). 

To address these problems, for the past three years work at the National Center 
for Research on Evaluation, Standards, and Testing CRESST at UCLA has focused 
on the following objectives: 

1. Design and develop a decision support tool, the web-based Quality School 
Portfolio (QSP) version 4.0; 

2. Develop a standardized training system for trainers and end users with 
certification for trainers; 

3. Implement a national rollout of QSP will multiple approaches to training 
and support; 
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4. Conduct evaluation research to evaluate the effectiveness of the features 
and functions of QSP, the effectiveness of the training, and the benefits of 
QSP use on educational improvement; 

5. Design requirements for the 5th generation of decision-support tools using 
results from the foregoing objectives. 

This report covers our work related to these objectives over a three year period. 

Objective 1: Design and Develop a Decision Support 
Tool, the Web-Based Quality School Portfolio (QSP) Version 4.0 

The design of the web Quality School Portfolio (QSP) version 4.0 was informed 
by CRESST's research on the implementation of the desktop version of QSP, in use 
in all 50 states, and by CRESST's research on testing, validity, data interpretation, 
and reporting strategies. Throughout the development of web QSP, empirical data 
from focus groups and beta testers in school districts across the country was used to 
systematically evaluate and revise the tool. 

This evaluation process helped identify bugs in the system and refinements 
that were needed, and it also highlighted the need for a number of features that 
were not included in the initial design of QSP version 4.0. For instance, as district 
and school personnel grew more knowledgeable about the requirements of the 
NCLB legislation they called for features that would increase QSP's capacity to 
monitor the progress of individual students. For example, not only did the users 
want to know which subgroups were not meeting standards, they also wanted to 
know which students comprised the subgroups and wanted access to individual 
students' records. In response, we added an individual student history report, which 
includes the capability to access data from prior schools, and also a feature in the 
Goals tab to view the students who are either meeting or not meeting the targets set 
for them. 

Description of QSP 

QSP is built with the latest technology, including Microsoft .NET development 
tools, and enables educators to bring together data from a range of disparate sources 
to analyze data as the basis for decision making at the school and district level. Core 
features of QSP are the longitudinal student database, the disaggregation 
capabilities, the report functions, the digital portfolio, and the goals and monitoring 
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functions. All QSP functions are organized by color-coded tabs, rather like a filing 
system. What follows is a description of the functionality included in each of the 
tabs that are used by educators: 

Student tab. Using QSP, educators can create an individual, longitudinal 
record for each student, including background information, achievement data, and 
data related to students' perceptions and opportunities to learn (Appendix A). For 
example, background information can include a student's age, ethnicity, and 
language spoken at home. Achievement data from statewide tests can be stored in 
QSP, and because QSP can be customized to meet the needs of specific districts or 
individual schools, data from district tests or tests that are only given at the school 
level can also be included. Information on students, teachers, and parents from 
surveys, interviews, and observations about other aspects of students' learning (e.g., 
feelings of safety in school, motivation, particular interests, opportunity to learn) can 
be added. With the Student tab, administrators and teachers can have easy access to 
each student's history to identify persistent problems and to monitor progress over 
time. 

A digital portfolio in which samples of student work and performance can be 
stored is accessed via the Student tab (Appendix B). The range of media used for 
storing artifacts in the portfolio, includes text, images, video, and audio. For 
example, a student's essay on the Civil War along with the scoring rubric for the 
essay might be included. A video of a student's presentation on a model he or she 
designed to illustrate a form of renewable energy, together with an image of the 
model could be stored in the portfolio. 

Groups tab. Test and other data can be disaggregated by various groups in the 
Groups tab (Appendix C). Users can create three types of groups: system, custom, 
and combination. System groups, which represent the first level of disaggregation 
(e.g., male and female), are automatically created in the system. Custom groups are 
created based on performance of students on various measures, including state, 
district, and classroom assessments (e.g., students scoring at the bottom quartile in 
math, represented as NRT Math <= 25). Additionally, users can combine system and 
custom groups to make a combination group. For example, teachers wanting to 
identify low-performing students on a number of measures could create one or more 
combination groups. First, they would create custom groups for each measure 
consisting of the lowest performing students (e.g., NRT Math <= 25, NRT Language 
arts <= 25, District Reading <= Below Basic, PBA Math <= 2, etc.). Next, they would 
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combine the custom groups, and at that point, QSP would locate the students who 
meet the criteria. Then the teachers would be able to see a list of the individual 
students who comprise the low-performing combination group across a number of 
assessments. 

Using the disaggregation capability in the Groups tab, educators can reveal 
hidden patterns in the data. For example, in a school with a high average score, it 
may be that sub-groups of students, including those in the at-risk group, are not 
doing well, but their performance is hidden because the higher performing students' 
scores are masking those of the sub-groups. 

The Groups function can also assist in program evaluation. In a situation where 
administrators want to determine the effectiveness of an intervention program, they 
could construct the variable, "participation in the program" and generate groups 
according to this variable. The performance of these students, pre- and post- 
intervention, could be compared. Also, a comparison of their performance with 
those students who had not received the intervention could be made. 

Reports tab. Once the groups have been defined in the Groups tab, reports can 
be generated to display the performance of the groups. QSP has 24 reporting options 
and users can select the most appropriate report for the analysis they are conducting 
(Appendix D). With QSP, users can conduct descriptive, comparative, correlation, 
longitudinal, and cross-sectional analysis. The results of the analysis are instant and 
are displayed in easily understandable formats. Reports are saved in PDF format, 
which are very small in size (10-12kb). This permits users to keep a large number of 
reports, to email them to colleagues and parents, and to use them in PowerPoint 
presentations for the school's community. 

Examples of the reports include: 

• a bar chart to show the performance by grade level on statewide tests 

• a pie chart to identify the ethnic composition of a low-performing group on 
the tests 

• a cohort line graph to show the previous years' performance of the low- 
performing group 

• a line graph to show the performance of students over time on benchmark 
assessments 

• a scatterplot to show the relationship between problem solving and 
computation on the schoolwide mathematics tests 
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• a floating bar chart to show the feelings of school safety for students 
performing at a range of proficiency levels 

• a relational report to show student performance relative to teacher 
characteristics (e.g., amount of professional development) and parent 
characteristics (e.g., education level) 

• a score range chart to show the range of scores for 9 th -grade students on 
statewide tests 

• a statistics chart showing the statistics ( mean, median, standard deviation, 
Ql, Q2, Q3, Q4, variance, confidence interval) for 6 th grade end of semester 
math test 

Goals tab. Using the goal function, educators can determine achievement goals 
and set targets to monitor student progress toward meeting standards (Appendix E). 
Each target represents the desired performance of a group of students on a 
particular measure over a specified time period. Targets can be mandatory or 
optional, allowing for conjunctive and compensatory models for meeting the goal. 
Once goals and targets are established, strategies that will be employed to realize the 
targets can be identified. One advantage is that schools are able to document their 
processes for meeting goals, and over time can review the effectiveness of the 
strategies they used. A timeline-based goals report can be generated to monitor 
progress on the targets, and through the "view +" and "view features in the Goals 
tab, teachers and administrators can see which students are meeting the target and 
which are not. 

Gradebook. There are two forms of gradebook in QSP. One functions like a 
traditional gradebook and permits percentage and letter-based grading. The other is 
a standards-based gradebook in which the teacher connects standards with 
assessments and scores the students' progress with reference to the standards 
(Appendix F). The gradebook variables can be linked to the core set of student data, 
which includes school-, district- and state-level testing scores, demographic, and 
opportunity to learn information. 

Additionally, teachers can quickly and easily create individualized student 
progress reports that show the up-to-the-minute status of students' performance on 
a range of classroom measures. The progress reports can be printed and sent home, 
or parents can be invited to view the progress reports online. 

In addition to the end-user tabs, there are a number of tabs that are for the 
exclusive use of system administrators. 
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• The District/ School tab provides the interface to manage districts' 
and/or schools' user accounts and search for users based on specific 
criteria. 

• The SysConfig tab permits system administrators to customize certain 
aspects of the system. 

• The In/ Out tab comprises all the import and export wizards in QSP. 

• The Database tab provides a complete view of all the student, teacher, 
and parent data in QSP and enables system administrators to manage 
the custom variables. 

• The Standards tab allows standards to be created manually, or imported 
from a file. 

Objective 2: Develop a Standardized Training System for Trainers 
and End Users with Certification for Trainers 

The development of data skills has not traditionally been a part of 
administrator preparation programs, and hardly ever a feature of teachers' pre- 
service and inservice training (Cromey, 2000; Stiggins, 2002), Moreover, professional 
preparation rarely includes training in school improvement processes using data 
(Cromey, 2000). Hence, providing educators with a data tool and with training in 
the software alone will not necessarily lead to effective data use. Educators also need 
training in data use skills. 

Equally important is training for IT personnel at the district and school level on 
how to set up the software, import data, and manage the database. Without skilled 
IT personnel, the chances of end users having access to reliable data are diminished. 

CRESST developed two training programs to meet the needs of educators (end 
users) and IT personnel. 

End-User Training 

The end-user training program was designed and first implemented in face-to- 
face environments. Revisions to the program were made based on the evaluations of 
participants. Subsequently, the program was transferred into an online environment 
and underwent the same process of implementation and revision. The online 
program comprises six online, self-paced, interactive modules and focuses on 
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training in the QSP software in the context of an investigative process that 
incorporates five core skills. 

The process for investigating data is shown in Figure 1. Embedded within each 
component of the process are the skills that we have identified to use data effectively 
for school improvement. 




Figure 1. Process for investigating data. 

What follows is a brief description of the skills included in each component of 
the process: 

What do we want to know? Questions start the process of investigation and 
determine the data that is collected. Different kinds of questions are asked 
throughout the process, often leading to further data collection. In our training we 
identify three types of questions, each of which has a different function. They are: 
general questions, drill down questions, and interpretive questions. In addition to 
the types of questions, we focus on the features of well-framed questions for data 
analysis, including: defining a clear purpose, what will be measured, how and when 
it will be measured, who will be assessed, and how the results will be interpreted. 

Collect data. Collecting data involves accessing existing data or collecting new 
data to answer the starting questions and the questions that arise from analyzing the 
data. Because collecting good data is key to getting credible information, our 
program focuses on the factors that influence data quality (e.g., validity, reliability, 
and alignment). 

Analyze results. Effective action is dependent on teachers' and administrators' 
capacity to analyze the data accurately and to infer a reasonable next step. The 
program concentrates on descriptive, comparative, and correlation analyses, the 
information that can be gained from each type of analysis, and on when to use 
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longitudinal and cross-sectional analyses. This module also focuses on which type of 
analysis matches each report in QSP. 

Set priorities and goals. After analyzing data, choosing priorities from among 
competing needs and establishing clear goals and benchmarks is key to guiding 
improvement efforts. We use Schmoker's (1999) criteria for creating goals: 
measurable; time-sensitive; focused on student achievement; linked to assessment; 
written in clear, direct language; and realistic and achievable. Once goals have been 
established, the program guides participants through a process of setting targets tied 
to assessments that have to be met for the goal to be achieved. 

Develop strategies. Knowing the "what" (the goals) without knowing the 
"how" (the strategies) for achieving the goals will not result in focused school 
improvement. The focus of this module is on establishing clear strategies that will 
define the specific actions needed to achieve the targets for each goal. 

Systems training. While the modules described above involve practice in using 
the functions of QSP in conjunction with learning data use skills, the program also 
includes a separate module that focuses exclusively on learning how to use the 
software. This module is self-paced, and users acquire information about the system 
from movies showing how each tab works and from written directions. There are 
also practice items for each of the tabs. An online help system and a glossary of 
terms are included in the software for users' reference after the training has 
concluded. 

Except for the systems training, throughout each module, participants respond 
to questions in their interactive journal. At the end of the module, there is a section 
termed "teamwork." Participants are given a copy of all their responses, and using 
the guiding questions provided, they can engage in individual reflection or get 
together with colleagues for discussion. 

Each module also includes a quiz, which covers the essential points of the 
training. If participants' answers are incorrect, they are given the correct answer 
after each item. They receive their total score at the end of the quiz. 

In addition to the online training, further support is available through the QSP 
User Manual. The manual comprises 10 chapters and provides comprehensive and 
detailed information about all aspects of QSP. 
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Technical Training for IT Personnel 

The technical training was developed using the same process as the end-user 
training and, in addition to the end user systems training, comprises four modules. 
What follows is a brief description of the content of each module: 

Server setup. This module focuses on the hardware and software requirements 
and on installing the QSP software. Participants learn how data is organized in QSP 
and how to prepare and import data. They also learn how to use the data parser in 
QSP (enabling them to convert ASCII type line data into a comma separated values 
file ready for import into QSP) and how to secure all the data. 

System administration. QSP has many different user accounts available, 
ranging from district superintendent, to principals, to program coordinators, to 
classroom teachers. This module focuses on how to set up all the user accounts. 

QSP IT systems training. In addition to the tabs described earlier in this 
report, a number of tabs are exclusive to IT personnel who administer QSP. This 
module introduces the tabs and provides instruction on how to use them. 

Data management. QSP administrators need to know how to use create, delete, 
backup, and restore databases, which is done through the database tab. In this 
module, participants learn how to use the database tab and how to manage the 
database. 

At the end of the program participants can print a technical manual, which 
gives them detailed information from each of the modules for future reference. 

Certification for Trainers 

The development of the certification for QSP trainers was undertaken in 
several stages. At the beginning of the national rollout of QSP, members of the 
CRESST project team conducted face-to-face training for end users and information 
technology (IT) personnel. Based on our experiences, we developed a set of 
competencies that QSP trainers would need to demonstrate in order to receive 
certification. Next, several prospective trainers, including district and school 
personnel, as well as consultants, attended QSP training sessions at CRESST, during 
which time project staff observed and evaluated their performance against these 
competencies. A number of these participants were certified as trainers and went on 
to train prospective users. 
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After these trainers had gained experience in working with districts and 
schools across the country, we convened a group of them to discuss standardizing 
the certification. In light of their experience of training, we revised the list of 
competencies for trainers, identifying specific skills in four areas: research and 
evaluation; data use; using the tool; and effective training. We devised case studies 
and short-answer questions to assess these competencies. 

Two of our training partners piloted drafts of this competency assessment in 
face-to-face sessions. Our last step was to make revisions to the certification 
materials as a result of the feedback we received from the face-to-face sessions. 

Through this process, we have standardized the certification of QSP trainers. 
To become certified, prospective QSP trainers are required to complete all of the 
online training modules described earlier, and answer both the multiple choice 
questions and questions about the case studies. The responses are evaluated by 
CRESST personnel. 

Objective 3: Implement a National Rollout of QSP With 
Multiple Approaches to Training and Support 

A phased national rollout of QSP began in January 2003. Phase 1, a pilot 
rollout, was conducted between January and April 2003 and was intended as an 
opportunity to test the stability of the QSP software, to learn about the import 
functionality and to do load testing with multiple concurrent users. The second 
phase of the rollout began in May 2003, and the final phase in September 2003. 

Those who participated in the pilot rollout were primarily drawn from existing 
desktop QSP users across the country. Districts were sent a letter of invitation, 
which included information about the server purchase and necessary licenses. 
Districts interested in participating invited one or more schools to be involved in the 
pilot. Before confirming their participation, to ensure districts of varying 
demographics, sizes, types, and locations in the pilot, as well as ensuring that they 
had the Internet access to use web QSP, we asked invitees to complete an online 
eligibility form requesting information about the schools and district and their 
technical capabilities. 

Regional training partners for the rollout were identified during the period of 
recruitment of districts for the pilot. These included independent trainers under the 
aegis of NCREL, County Offices of Education staff, state training teams, district 
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superintendents, and district leadership teams. They were invited to attend QSP 
training by CRESST personnel at UCLA and became the trainers for the pilot phase. 

After the pilot phase was completed, new districts were "rolled" in during 
phase 2. These comprised districts that had attended CRESST presentations on QSP, 
and also a number that had become aware of QSP, independent of CRESST. In the 
main, existing training partners conducted the training for phase 2, but some district 
leaders did attend trainings by CRESST both at UCLA and at sites in their region. 

Phase 3 of the rollout began in September 2003 and continued until September 
2004. For the most part, districts in this last phase approached CRESST to participate 
in the QSP program. Although CRESST personnel attended some informational 
regional meetings by invitation, most information sharing about QSP was done via 
web demonstrations. During this period we expanded our cadre of training partners 
to include additional district trainers, independent consultants, and foundation 
personnel. 

Throughout each phase of the rollout, a range of approaches to training and 
support were adopted. These were: 

• Attendance at a two- or three-day CRESST training with follow-up technical 
support by CRESST personnel. 

• Attendance at a district training conducted by district trainers (length of 
training varied, but often one day) with follow-up support from the district. 
In many cases districts initially produced reports for the schools with the 
goal of eventually transferring responsibility to the school site. IT support 
was regularly given by CRESST to district personnel. 

• Attendance at a district training conducted by district trainers, with 
continued support on the technical aspects and data querying given by 
designated district personnel. This support often involved school visits. 

• Multiple days training throughout the year from independent consultants 
(certified by CRESST), using CRESST's core program, but tailored to 
individual districts. Ongoing technical and data querying support at the 
school site was given by the consultants, and in some cases, data cleaning 
and importing was done by the consultants. 

• Online CRESST IT and data querying training without follow up support. 
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Objective 4: Conduct Evaluation Research to Evaluate the Effectiveness of the 
Features and Functions of QSP, the Effectiveness of the Training, and the Benefits 

of QSP Use on Educational Improvement 



Background 

Research points to the importance of data analysis for school improvement 
(Anderson & Postl, 2001; Cawelti & Protheroe, 2001; Cotton, 1992; Glickman, 1998; 
Johnson, 1997; Khanna, Trousdale, Penuel, & Kell, 1999; Lee, Herman, & Mitchell, 
2001; Rubenstein & Wodatch, 2000; Teddlie & Reynolds, 2000; Tognieri & Anderson, 
2003). Additionally, longitudinal analysis of student and subgroup performance can 
help practitioners to identify strengths and needs in curriculum and instruction and 
potentially lead to school improvement (North Central Regional Educational 
Laboratory, 2000). Moreover, analysis of benchmark assessment data, in particular 
when conducted by teachers and principals who have received training in 
interpreting these data, can help raise students' performance (Snipes, Doolittle, & 
Herlihy, 2002). 

CRESST research on data use for continuous school improvement utilizing a 
data tool (the desktop version of QSP) reported that schools started to look at data in 
ways they had never done previously and came to a deeper understanding about 
the levels of performance among their students. The study also found that 
examining data promoted dialogue in schools about raising achievement (Lee et al., 
2001; Mastergeorge, Hammersley, Fennell, & Barela, 2001). Other research on data 
analysis using desktop QSP found that some schools successfully undertook 
longitudinal analyses of assessment data, which led to changes in programs and 
practice, while others analyzed "event-based" data and used the information for 
discussions on student behavior and teacher management practices (Webb, Mason, 
Choppin, Green, Thom & Watson, 2001). 

While there is little to no pre-service emphasis on the use of data in school 
improvement processes (Cizek, 2000; Cromey, 2000; Stiggins, 2002), research 
suggests that providing in-service training for school personnel to use existing data 
to analyze both student achievement data and non-academic data, and providing 
instruments to encourage local data collection can have a positive impact on school 
improvement and student performance (Anderson & Postl, 2001; Khanna et al., 1999; 
Rubenstein & Wodatch, 2000; Ward, 1998). Additionally, helping schools formulate 
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their own research questions and collect local data can lead to more accurate 
information about students and to better decisions in school improvement efforts 
(Glickman, 1998). 

Rationale for QSP Evaluation 

Prior research provides evidence that collecting, analyzing, and interpreting 
school data on a timely basis, and supporting educators to use data through in- 
service training, can lead to higher student achievement. Additionally, research on 
the desktop version of QSP foreshadowed the potential for school improvement 
afforded by a web-based version of QSP. This evaluation research focused on 
practitioners' data use with QSP. Specifically, the study examined how educators 
used web-based QSP, how QSP assisted the data collection and analysis in schools 
and districts, and the impact of QSP use on school improvement efforts. Data that 
examined other factors expected to affect school improvement efforts and QSP 
implementation were also collected. These factors include professional training and 
effective communication between school leaders and teachers (Hipp & Huffman, 
2000; Newmann, King, & Rogdon, 1996; Wheeler & Agruso, 1996). 

Research Questions 

The evaluation research was designed to address three main questions: 

1. How is web-based QSP being implemented at different sites? 

2. What affects the QSP implementation process? 

3. How does using web-based QSP affect schools' and districts' data use and 
decision-making? 

Evaluation Methodology 

This evaluation employed both qualitative and quantitative research methods. 
Evaluation data was collected through: a) a post-QSP user survey (N=59, response 
rate 45%); and b) in-depth telephone interviews with a further sample of 41 current 
users, who were not in the survey sample. 

Sites and Participants 

The participants in this research were from a range of school districts (rural, 
suburban, urban) in 15 states across the country and were all voluntarily 
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implementing web QSP. Two post-QSP surveys were developed. The long post-user 
survey was designed for those who had begun implementation and continued using 
QSP. Those who had begun implementation and were stalled or had discontinued 
were asked to complete a shorter survey. Altogether, we received 59 completed 
post-QSP user surveys, 30 from those who had continued to implement QSP and 29 
from those who had either stalled or discontinued their implementation. An 
additional 41 respondents, all active web-QSP users at the time of contact, were 
interviewed by telephone. 

Measures: Post-QSP User Surveys 

The long post-QSP user survey, completed by those who were using QSP, 
collected data in the following areas: 

User background characteristics. Data were collected about users' technology 
experience and attitudes toward data use. 

Data use. Users were surveyed to measure their level of QSP use, including 
how frequently and to what extent they a) used different functions of QSP; b) 
conducted data collection; c) conducted data analysis; d) received support at the 
school and district level for QSP implementation; e) felt the impact of QSP on their 
use of data to make decisions about students' learning; and f) communicated 
information about students with colleagues and parents. 

QSP training and professional support. QSP training environments included 
online training, face-to-face training, and a mixture of both, two modules online and 
the remaining modules in a face-to-face context. Users were asked to comment on 
who conducted their training, in what environment, and how useful it was in 
preparing them to implement QSP. They were also asked about the extent of 
continued support they have for QSP. 

The short post-QSP survey, completed by those who had either stalled or 
discontinued using QSP, collected data on: a) the nature of their training, b) their 
reasons for participating, c) features of QSP that they found useful and not useful, 
and d) reasons for discontinuing their implementation of QSP. 

User Telephone Interviews 

Phone-interviews, lasting 30-45 minutes, were conducted with a further sample 
of current users (N = 41). All the interviews were recorded. The interviewer asked 
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the participants to comment on their experience of using QSP, what they hoped to 
achieve with QSP, the effectiveness of the training sessions, the support they 
received at the school and district level for QSP implementation, and the impact of 
QSP on their use of data to make decisions about students' learning. During 
interviews, users were also prompted to reflect on how data collection, analysis, and 
reporting using QSP has affected teacher collaboration, communication among 
teachers, communication between principals and teachers, and communication 
between teachers and parents. 

Survey analysis. The answers to the open-ended survey questions were 
analyzed qualitatively along with the interview data. Descriptive analyses were 
conducted on the survey data, which yielded frequency counts or means for the 
following survey items: a) technology use; b) use of QSP features and functions; c) 
data use; d) QSP impact; and e) training and professional support. 

Qualitative analysis. Each interview tape was numbered according to the 
sequence of the interviews, and all the interviewees were given pseudonyms to 
ensure their privacy. All the interview tapes were transcribed and coded to examine 
emerging patterns and themes that illustrated the implementation process, 
experience with training, and the perceived impact of QSP. 

A list of codes was created prior to the interviews based on the research focus 
and key concepts of the evaluation. New codes emerged during the interview and 
analysis process. For reporting purposes, the coded data were sorted according to 
major themes or categories. 

Research Results 

The majority of our respondents to the long and the short surveys (81%) were 
school or district administrators, as were those interviewed by telephone (91%). The 
tables below report data from the long survey. Data from the interviews and 
summary data from the short survey are included in the narrative. 

Technology access. Most respondents had access to a computer at work with a 
high-speed Internet connection. All of them had a computer at home and 58% had 
access to a high-speed Internet connection at home. However, while all respondents 
stated that they used word-processing and email software on a daily or weekly 
basis, there was a relatively steep decline in the use of more quantitatively oriented 
software packages. Whereas 71% used a spreadsheet package such as Microsoft 
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Excel on a daily or weekly basis, this figure declined to 25% in the case of database 
software, and to 8% for statistical packages such as SPSS (see Table 1). 



Table 1 



Users' Technology Access and Use 



Technology information 


Percentage of 
users 


Having a computer at home 


100% 


Having access to shared computers in school 


80% 


Having access to high-speed Internet 
connection at office/ classroom 


88% 


Having access to high-speed Internet 
connection at home 


58% 


Daily or weekly spreadsheet use 


71% 


Daily or weekly use of database software 


25% 


Daily or weekly use of statistics package 


8% 



Reasons for implementing QSP. Most users expected to use QSP for data 
analysis (73%), with the least number hoping to improve their classroom instruction 
with QSP (4%). The low response for the latter is compatible with the fact that the 
vast majority of respondents were administrators, rather than teachers. However, 
the teachers who did participate in the survey may also not yet see the possibilities 
of data analysis for improving classroom instruction. Half of the respondents 
intended to demonstrate student achievement with the reports, while 41% indicated 
that their participation in QSP was motivated by the instructions of supervisors. A 
lesser number (32%) reported that they wanted to collect and store data for 
themselves (see Table 2). 



16 




Table 2 



Participants' Reasons for QSP Implementation 



Reasons for QSP implementation 


Percentage of 
respondents 


I planned to use QSP to help me conduct data analysis 


73% 


I intended to demonstrate student achievement with 
reports from QSP 


50% 


I was instructed by my school to use the software 


41% 


I wanted to collect and store data myself 


32% 


I hoped to improve my classroom instruction with QSP 
use 


4% 



Access to Data 

Lack of easy access to data is an obstacle to educators' use of data for school 
improvement (Thorn, 2001; Wayman et al., 2004). Interview participants reported 
that QSP had improved their access to data and had resulted in increased data use. 
For example, one superintendent commented: 

To be quite honest, I have done much more with data and data analysis the last year than 
I did probably the three previous years in the district just because of the ease of use. It's 
very easy to use software. It's step-by-step. Also the data is all in one place. In the past, it 
may have been all in my office, but maybe in four different file cabinets. To have to do 
that by hand as opposed to being able to do it electronically, it has saved me hours and 
hours, if not days of time. 

A school principal pointed out that before having access to QSP, his school only 
had an attendance database, which lacked the capacity to store student achievement 
data. Student achievement scores had been kept on paper, which impeded data 
analysis. With access to QSP, for the first time, student academic data was being 
stored electronically, and he envisioned that QSP would enable him and his 
colleagues to compare student performance from different years and to be able to 
examine trends in performance: 

We have an attendance database called FASE...But there is no room on that program to 
input the academic stuff. . .We're depending on QSP to be our database. . .This is why I'm 
saying that we're taking a big leap because we have not been technologically 
sophisticated. ... We have this year's data, and now next spring when we test again, we'll 
have next year's data, and I'll be able to compare next year to this past year and things 
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like that. . .the more years that we build on it, I think the better off we're going to be able 
to look at some long-term trends of what's happening with our kids. 

For a seventh-grade reading teacher, access to data meant that she could 
readily use the test item level information to guide her instruction: 

The obvious strength is you have that information right at your fingertips. I mean it's 
right there and you're not having to go through each child's file individually.... Because 
the test scores that we put in, it gives us a cross section of not only reading 
comprehension but just basic understanding, vocabulary, and all of that kind of stuff. 

And so it allows us to have more specific goals in mind of what each child needs 
individually, and as a whole population. 

However, the short survey data and the interview data also point to some 
important factors that need to be considered before access to data can be achieved. 

School data can exists in many different places, ranging from a main district 
data store to Excel spreadsheets, to loose papers in files. Prior to data entry, it is 
important to take an inventory to identify data sources and locations and from this 
inventory determine which data will be useful for analysis and will be imported into 
the data tool (Wayman et al., 2004). An equally important task is to make sure that 
the data are "clean." Inaccurate and incomplete data will present problems for data 
analysis and so educators will need to assess the quality of the existing data and take 
the necessary steps to ensure clean data. Finally, it will be important to identify the 
resources for any data cleaning and for importing data into the system. 

These data preparation tasks described above had a negative impact on 
attitudes toward QSP use. A number of interviewees stated that prospective users' 
initial enthusiasm for using data with QSP was dampened somewhat when they 
realized what was required to clean their data before importing them into the 
software. Most of them had not been aware of what was involved in data 
preparation. A principal summed up a central problem when he stated: 

The hardest part was getting the data into the system. The reason being because we 
didn't really have a protocol here for preparing data within our student information 
system. For example, when the kids took the PSSA, their student number wasn't on the 
test. So we had to mix and match all that stuff up. 

Other users indicated that the amount of available IT personnel time is a factor 
that can impact data use. At the beginning of implementation, sites were importing 
data for multiple years, which required a great deal of time. An additional time 
burden resulted from the iterative process of software development described 
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earlier. New versions of QSP were released periodically, which required data to be 
re-imported. However, even with the final version of QSP, because importing data is 
an ongoing process, longer-term considerations of time remain. For example, one 
user pointed out that he is importing state level data several times a year: 

I'm continually adding new things because our state information comes sometimes 
quarterly, as far as the demographic data. The testing data usually comes in the spring. 

So when that stuff gets here, I run it through all of our programs and then run it into 
QSP. 

Those districts and schools using additional kinds of data (e.g., from 
schoolwide tests and benchmark assessments) are required to import data 
frequently and may lack the resources to do that. One interviewee, who was 
involved in an incremental process of district-wide implementation, was concerned 
about whether current resources would be sufficient when the whole district 
implemented QSP. 

Reponses to the short survey (those who had stalled or discontinued 
implementation) indicated that the most common reasons for discontinuing use 
were the amount of time for data preparation and importing data into the system. 

One possible solution to the time and resource problem came from a 
consortium of school districts in Michigan who had a central administrative service 
that is responsible for data entry and technical support, with each district paying an 
administrative fee. This enabled districts that would not be able to afford this kind 
of support if functioning on their own to pool resources and make data analysis 
available to their schools. 

Another factor highlighted by several interviewees was that before preparing 
and importing data it is important to consider the types of data available and which 
data the will be most useful. Based on his experience of importing data, one 
interviewee stated: 

The thing I would do first is to just formulate in your mind and on paper the types of 
data that you have, just so you can start to visualize what you want to get in there. From 
there, I would ask myself which sets of these data can I get or which ones do I have or 
how hard would it be to get them. 

A district administrator reported that after a period of QSP implementation, he 
had reviewed the value of the existing data in the system with administrators: 
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We asked last spring how to make the data set that was in QSP more valuable. With that 

feedback we have been working this fall to add all of those items. 

Climate for Data Use 

Although QSP had increased access to data in schools, it will remain largely 
ineffective if teachers and administrators are unwilling to use it. For practitioners to 
develop a commitment to data use, district and school climates that trust data and 
support data use must be nurtured. The survey respondents were prompted to 
express agreement or disagreement with statements that described factors 
contributing to the climate in their schools and districts for data use. Overall, the 
results showed that there was a positive climate for data use among respondents. 
Ninety percent reported that value was placed on data use in their schools and 
districts, 70% of them thought that effective decisions were made based on data 
analysis, and 80% reported that data use is an expectation at their site. Seventy-five 
percent of respondents knew where to seek help in interpreting student 
performance, and 70% reported that there were opportunities to receive training on 
data based inquiry at their school. Trusting the data is a pre-requisite for effective 
data use, and significantly, 70% of respondents reported that they trusted the data in 
the system. Also of significance is that 65% of respondents reported that very few of 
their colleagues had experience working with data, an indication of the extent of 
professional training still required (see Table 3). 
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Table 3 



Climate for Data Use - Frequencies 





Percentage of respondents 


Climate for data use 


Strongly 




Strongly 


agree/ agree 


Not sure 


disagree/ disagree 


The decision makers at my school/ district value the 
information gained from data analysis 


90% 


10% 


0% 


The decision makers at my school/ district make effective 
decisions based on information gained from data analysis 


70% 


25% 


5% 


1 can't trust the data in our school system 


15% 


15% 


70% 


It is expected at my school that we use data on a regular basis 


80% 


10% 


10% 


Very few of my colleagues have experience of working with 
data 


65% 


10% 


35% 


The principal encourages use of data in instructional planning 


20% 


15% 


65% 


I know where to seek help if I don't understand how to 
interpret student performance data 


75% 


15% 


10% 


My school provides teachers the opportunity to receive 
training on data based inquiry 


70% 


10% 


20% 



Respondents were also asked about the extent to which their schools and 
districts provided professional development and support for data use. A majority of 
respondents reported that data use was supported through professional 
development. However only a minority believed that their schools and districts had 
the technical support to use data effectively, and even fewer believed that their 
school or district had the capacity to use data effectively without such support. 
These results are displayed in Table 4. 
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Table 4 



Professional Development and Support from Schools/ Districts 



Our school/ district: 


Strongly agree/ 
agree 


Not sure 


Strongly disagree/ 
disagree 


Provides professional development support for 
data use. 


58% 


21% 


23% 


Has been able to get the data it needs. 


56% 


28% 


17% 


Has received professional development provided 
by trainers that has been helpful in using data. 


58% 


21% 


21% 


Gets the technical help we need to use data 
effectively. 


47% 


32% 


16% 


Has the capacity to use data independently 
without outside support. 


37% 


26% 


37% 



Taken as a whole, these results suggest that despite a cultural climate that is 
generally favorable to data use in the school districts we surveyed, reservations exist 
about the capacity to use it effectively. These reservations are quite commensurate 
with the respondents' levels of experience with numerical data beyond the 
spreadsheet level documented above. 

Features and Functions of QSP 

The core features of QSP are groups and reports (see Section 1 for a full 
description) and were the most frequently used by all users. In the main, 
respondents were using the disaggregation features in the Groups tab, and were 
generating reports on the performance of these groups. Although many interview 
and survey respondents were not fully using the longitudinal capabilities, they 
indicated that this was a feature they liked in QSP. Those who were using this 
function had examined state data over a number of years, looked at trends over 
time, and followed cohorts. 

Our survey and interviews asked about the extent to which respondents were 
using the gradebook and digital portfolio functions of QSP. However, because these 
functions are designed for use in the first instance by teachers, and few of the latter 
participated in the study, data on the use of these functions is sparse. While several 
districts had begun to train a cohort of their teachers in the gradebook, many 
districts had already implemented an electronic gradebook and, because of the 
investment of time and money in the implementation, they were not inclined to use 
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the QSP gradebook. A small minority of respondents had begun to use the goals 
feature in QSP. Least used by our respondents was the digital portfolio. While a 
number of people were very positive about the inclusion of the digital portfolio to 
store performance data, they were not yet at the point where they had these kinds of 
data available, although several indicated that it was a future goal. 

Significantly, a considerable majority of respondents to the short survey (those 
discontinuing QSP implementation) had positive comments about features of QSP 
that they found useful, ranging from the querying and disaggregation capabilities, 
to easy-to-generate reports and the clarity of their graphics, to the gradebook. A 
minority of the respondents found the software difficult to use and not user friendly, 
and several suggested that it would be easier to use if pre-determined questions and 
pre-built reports were included. Suggestions for improvements in the features and 
functionality of QSP that emerged from all data sources are included in Section 5 of 
this document: specifications for a 5 th generation of QSP. 

Types of Data 

The long survey prompted users to indicate the frequency with which they 
used the following data: student achievement data, student background data, 
opportunity to learn data, and perception data. 

Achievement data. State or district achievement data was used most 
frequently (75%), with only 10% of respondents using them on an annual basis and 
15% never using them. This suggests that these large-scale achievement tests are 
being analyzed more frequently and that district data (for example, benchmark data, 
which districts are increasingly using) are being used more formatively and as 
predictors of achievement on statewide tests. These uses were confirmed by a 
number of interviewees. Student grades were used quarterly or more often by 75% 
of respondents, and 25% never used grades. Diagnostic or placement tests were also 
frequently used by 70% of respondents. The relatively frequent use of diagnostic or 
placement tests also suggests that these are being used formatively to plan 
instruction and to place students in classes or programs appropriate to their needs. 
Teacher-developed assessments were used quarterly or more frequently by 60% of 
respondents, with 25% never using them, and curriculum embedded assessments 
were used frequently by 55%, while 33% of respondents never used them (see Table 

5 ). 
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Table 5 



Data Use: Achievement Data - Frequencies 



Achievement data use 


Percentage of 
respondents 


Percentage of 
respondents 


Percentage of 
respondents 


Quarterly or 
more frequently 


Annually 


Never/no access 


State or district achievement data 


75% 


10% 


15% 


Diagnostic or placement tests 


70% 


5% 


25% 


Curriculum-embedded 


55% 


5% 


40% 


Student grades 


75% 


0% 


25% 


Teacher-developed classroom assessments 


60% 


25% 


15% 



Background data. The survey showed that 65% of participants used student 
attendance data frequently, 10% used them annually, and 25% never used them. 
Student behavior data was used frequently by 60% of participants, annually by 10%, 
and never used by 25% of respondents. Sixty percent of respondents used student 
demographic data frequently and relative to the previous data items more 
respondents used them annually (25%), while a smaller number never used these 
data (15%). Student preschool participation was frequently used by only 15% of the 
respondents; 25% used them annually; and 60% never used them at all. Student post 
graduation plans were least used frequently (10%), 25 % used then annually and 
they were never used by 60% of respondents (see Table 6). 



Table 6 



Data Use: Background Data - Frequencies 



Background data 


Quarterly or 
more frequently 


Annually 


Never/no access 


Student attendance 


65% 


10% 


25% 


Student demographics 


60% 


25% 


15% 


Student behavior 


60% 


10% 


25% 


Student preschool participation 


15% 


25% 


60% 


Student post graduation plans 


10% 


25% 


65% 



Opportunity to learn. Program participation data was used frequently by 45% 
of respondents, used annually by 25%, and never used by 21%. Thirty-four percent 
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of respondents used resource allocation data frequently, while 50% never used 
them. Course taking information was used frequently by 33% of respondents, 
annually by 4%, and never by 36%. Time spent on subject data were never used by 
58% of respondents, used annually by only 8%, and more frequently by 17%. 
Similarly, content exposure data was never used by 54%, used annually by 12%, and 
frequently by 17% of participants. Finally, data about teachers' qualifications was 
never used by 46% of respondents; 25% used them annually and 12% more 
frequently (see Table 7). 

Table 7 

Data Use: Opportunity to Learn - Frequencies 



Percentage of respondents 



Opportunity to learn 


Quarterly or 
more 

frequently 


Annually 


Never/no access 


Program participation (ELL, Special 
Ed.) 


45% 


30% 


25% 


Resource allocation 


40% 


0% 


60% 


Courses taken 


40% 


5% 


55% 


Time spent on subjects 


20% 


10% 


70% 


Content exposure 


20% 


15% 


65% 


Teacher qualification 


15% 


30% 


55% 



Attitude and perception data. The survey showed that 50% or more of 
respondents never used attitude and perception data. Respondents mostly used 
these data on an annual basis. Student attitude and parent and teacher perception 
data were used annually by 40% of respondents, and 35% used student perception 
and administrator perception data annually. Student attitude, parent, teacher, and 
administrator perception data were frequently used by 10% of respondents, while 
15% used student attitude data frequently. The results are displayed in Table 8 
below. 
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Table 8 



Data Use: Attitude and Perception - Frequencies 



Attitude and 
perception data 


Quarterly or more 
frequently 


Annually 


Never/no access 


Student attitude 


10% 


40% 


50% 


Student perception 


15% 


35% 


50% 


Parent perception 


10% 


40% 


50% 


Teacher perception 


10% 


40% 


50% 


Administrator 


10% 


35% 


55% 


perception 









NCLB requirements have led to a greater focus on data, but our results suggest 
that many practitioners restrict themselves to analyzing student characteristics in 
relation to achievement data. The vast majority of examples of QSP use described by 
interviewees in the telephone survey also reflected the prevalence of this kind of 
analysis. While respondents, in conformity with the provisions of the NCLB 
legislation, are using QSP effectively to examine whether racial, economic, and 
linguistic disparities in achievement are narrowing or not, as yet they have not 
moved to more complex analysis of the factors that might be contributing to 
differential performance levels. While they may want to move beyond their current 
level of analysis, they may not yet understand the possibilities for creating data 
elements and indicators This is particularly underscored in Table 7, which shows 
that the majority of respondents do not use opportunity to learn data such as time 
spent on subjects, teacher qualifications, and content exposure, all of which could 
potentially influence achievement. Moreover, the majority of interview and survey 
respondents were using reports that embodied descriptive, rather than inferential, 
statistics. For example, the most frequently used report format was the bar chart. A 
considerable number of survey respondents indicated that they did not yet feel 
"comfortable" or "ready" to move to more complex analysis and reporting. Indeed 
one superintendent, experienced in data use, stated that members of his district 
management team were only using bar charts: 

They're using bar charts because, for the most part, they don't understand the correlation 
the way I do. They're not sure even after the training we've done. They're not all together 
comfortable with the kinds of analysis I do. 
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Who Analyzes Data? 

Survey data showed that most of the data analysis was being conducted within 
schools or districts primarily by administrators, a fact which was further reinforced 
by the interview data. 72% of survey respondents indicated that school 
administrator did the analysis, 61% reported that the school leadership team 
analyzed the data and 56% responded that analysis was performed by district 
administrators. A combination of school administrators and teachers (50%) were the 
next largest group with only 44% of respondents indicating that teachers were 
involved in analysis (See Table 1). Although the number of teachers conducting data 
analysis is less than administrators, given that teachers traditionally lack training in 
data analysis, and that analyzing data with technology tools is relatively new to all 
school personnel, the fact that 44% of participants responded that teachers analyzed 
data can be construed as a promising sign (see Table 9) . 

Table 9 



Who Analyzes Student Performance Data? 



Who analyzes performance data? 


Percentage of 
respondents 


School administrator 


72% 


School leadership team 


61% 


District administrator 


56% 


School admin/ teacher collaboration 


50% 


Teachers 


44% 


We send the data out 


17% 


Other 


11% 


No data analysis 


6% 



To Whom Are Data Regularly Reported? 

A considerable majority of participants (89%) responded that the data were 
regularly reported to teachers. This would suggest that the reports were intended to 
influence instructional practices. A number of participants in the interviews 
reported that they had given reports to teachers to assist them in meeting students' 
needs. For example, a county office consultant for data use described how one high 
school was looking at reports of the students' performance: 
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I helped the principal at the high school generate lists for his ninth, tenth, and eleventh 
grade to share at a faculty meeting. We were able to print out all of the kids who were 
not proficient in English/ Language Arts and in Math. They were a school that really 
wasn't accessing their Star Data very well. We were able to pull up who these kids are 
and what their scores are. Are they close? Are they not close? He gave that to the teachers 
and said, "This is a group of kids that we need to be aware of." 

At another school, the principal reported that she used QSP to generate a 
master list of all students in her school whose performance was below standards. 
After she identified the students who needed help, she was able to use QSP to pull 
out their individual math and reading scores on state tests. Once the data set of 
children with specific needs for each grade level was created, the principal 
presented the list and the QSP reports to the teachers. The student list, their test 
scores, and the QSP reports provided valuable information for the classroom 
teachers to focus on students' needs and to plan instruction tailored to individual 
students. The principal stated: 

I was able to identify the children plus have their achievement level scores readily 
available in the list. With CTBS, all I would've been able to do is generate their scores, 
and then I would've had to go to our student information systems and pull out the 
children that fall under No Child Left Behind, and then come back and make a different 
list on my own. ... We have to know who they are, and we have to monitor them. 

The survey also showed that data were reported frequently to parents by 74% 
of respondents, to state agencies by 74%, with 68% reporting data to the district, and 
58% to the school board. Data were reported regularly to students by 53% of 
respondents. Strikingly, only 26% of respondents reported data to feeder/ receiver 
schools. Data were reported least to funding agencies (16%), and also 16% indicated 
that they reported data to business and community groups (see Table 10). From the 
open-ended question on the survey about why data were reported, among the most 
commonly cited reasons were showing school progress, NCLB and other reporting 
requirements, and guiding instructional decision making. 
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Table 10 

To Whom Are the Data Regularly Reported? 



To whom are the data regularly reported? 


Percentage of 
respondents 


Teachers 


89% 


Parents 


74% 


State agencies 


74% 


District office 


68% 


School board members 


58% 


Students 


53% 


Feeder/ receiver schools 


26% 


Funding agencies 


16% 


Business/community groups 


16% 


Other 


10% 



Purposes of Data Use 

The survey revealed that the main purposes of data use were to measure 
student progress and program effectiveness, to guide curriculum and instruction, 
and to meet accountability reporting requirements: 68% of respondents used data to 
measure student progress and 68% to measure program effectiveness. The majority 
of respondents (74%) used data to guide curriculum development, 68% to meet 
accountability reporting requirements, and 63% respectively to assess instructional 
effectiveness and to plan individual student's learning. Fifty-eight percent of 
respondents used data for strategic planning. Data were least used for placing 
students (42%), for allocating resources (37%), and for guiding professional 
development (47%) (see Table 11). 
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Table 11 



Main Purposes of Data Use 



Main purposes of data use 


Percentage of 
respondents 


Measuring student progress 


68% 


Measuring program effectiveness 


68% 


Assessing instructional effectiveness 


63% 


Planning individual student's learning 


63% 


Guiding curriculum development 


74% 


Meeting accountability reporting requirements 


68% 


Student placement 


42% 


Showing trends 


63% 


Allocating resources 


37% 


Strategic planning 


58% 


Setting goals 


63% 


Making changes to curricula and programs 


68% 


Guiding professional development 


47% 



Purposes for using data uses that correspond to several survey items emerged 
from the interview data. Six interviewees stated that their sites are performing 
analysis with QSP for the purpose of accountability reporting requirements of 
individual states and NCLB. A principal commented that: 

Since No Child Left Behind, you know we have to look at data, and we have to look at 
achievement as it relates to anybody who is covered under No Child Left Behind. So I 
was able to use QSP and come up with a master list of all students in my building who 
fall under one or more categories of the No Child Left Behind categories. Then I was able 
to include in that their math and reading scores, which is what is used to determine our 
adequate yearly progress. 

Six interviewees stated that they are using QSP to evaluate the effectiveness of 
programs. For example, one superintendent described how in his district they were 
using QSP to evaluate two reading programs: 

We have Houghton Mifflin that we have been using for the last 8 or 9 years probably. 

And we have one school in the district that is a Reading First school, and they are 
piloting Open Court. We haven't figured out how to control for all of the other services 
that come with Reading First, but we are looking at the amount of gain and the 
performance of students who are in the Open Court system versus Houghton Mifflin. 
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Another district administrator recounted how, by using QSP, his school district 
administrators were able to track students when they left middle schools to go to 
high schools. Student performance data in the high school was also sent back to 
middle schools so that middle school teachers could see the long-term impact of the 
intervention programs they provided: 

We're trying to do some program evaluation, and we're able to code kids that might be 
in that program and track them. For us, it's making a difference because we can look at it. 
Instead of saying, "I have a gut feeling about this," now we're actually able to look at the 
data. 

A common theme among interviewees was using data to plan for students' 
learning. Four interviewees described how data analysis with QSP was leading to 
more differentiated instruction. For example, a principal stated: 

For the most part we were getting kids to proficiency the same way you get them to 
Broadway: practice, practice, practice. And we've kind of run into the brick wall of 
diminishing returns with that approach, so now what we're looking at is using data 
analysis and more online real-time testing in order to be able to differentiate instruction 
for those various students so that we can focus on their particular strengths and 
weaknesses, particularly their weaknesses, and try to remediate those so that we can 
bring them up to proficiency. 

Another principal noted: 

We tell our teachers that they need to be grouping students by the subtests like what 
students need a particular reading skill. So we're able to group students by that. We're 
also able to know what students need more help for remedial programs, and what 
students just need some coaxing because they're close to the target. So I would say 
predicting what kind of programs or what kind of special help we need to give them. 

We've used it for that. 

Data from another 10 interviewees showed that QSP was being used to 
measure students' progress and to identify students who were at risk of not meeting 
standards so that curriculum and instruction could be adapted to meet their needs. 
One principal stated: 

Basically we've been using it to determine — specifically looking at students who are not 
proficient, digging around and drilling down to find out what we can do as far as 
remediating them. That is basically where we've focused our attention. 
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At another school, at-risk students were identified using QSP, and the school 
provided extra help to those students during after school hours. A superintendent 
described QSP's applicability to after-school interventions: 

We have something that we call "Targeted Services." It's some things that we do after 
school for our students. They [building administrators] have utilized QSP to basically 
generate some of the lists of those who are at risk of not being successful. That has 
helped them to extend to parents the invitation to, "Would you like to be involved in this 
extra activity to help strengthen some of those skills that your children may not have?" 

Some districts and school had begun to use QSP to set goals and to monitor 
their student progress. One superintendent revealed that he relied on QSP to initiate 
discussion with his staff about student improvement efforts. 

I laid that (student performance) out on that chart to say "Is this where we want our 
second graders to be?" Is this where we want our ninth graders to be?" We've begun to 
have those discussions, which I think will lead to more when we meet at a retreat with 
our Board for our continuous improvement processes to set different goals, and then 
utilize QSP to see the progress we're making on them. 

None of the interview respondents reported that they had used QSP for the 
purposes of student placement, allocating resources, or for guiding professional 
development. This is broadly consistent with the survey results where the data show 
these purposes were least used for data analysis. 

Impact of Data Use 

Notwithstanding the relatively limited range of uses for data, the respondents 
clearly indicated that data use had significantly improved a number of facets of 
school and student evaluation. In particular, respondents indicated that the 
improved availability of data had led to a stronger focus on student performance 
(63%), an enhanced capacity to evaluate special programs (55%), and increased 
attention to the performance of student sub-groups (53%). A substantial minority 
also suggested that the greater availability of data had enhanced the alignment of 
curriculum assessment and student performance (47%), as well as the ability to 
revise instruction (42%) and curriculum (38%). Encouragingly, 42% stated that QSP 
has increased their enthusiasm for data. The results are displayed in Table 12. 
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Table 12 



Data Use Impact on School and District 



QSP has improved our school's/ district's: 


Strongly agree/ 
agree 


Not sure 


Strongly disagree/ 
disagree 


Data availability 


69% 


17% 


16% 


Focus on student performance 


63% 


21% 


16% 


Ability to evaluate special programs 


55% 


22% 


22% 


Attention to performance of student sub-groups 


53% 


26% 


21% 


Alignment of curriculum assessment and student 
performance 


47% 


36% 


16% 


Ability to revise instruction 


42% 


37% 


21% 


Enthusiasm for data 


42% 


32% 


26% 


Ability to revise curriculum 


38% 


42% 


21% 



One superintendent noted that the impact of increased data availability had 
encouraged more data sharing among his staff: 

We've had to collect data before, but the problem was it was not something that 
everyone could actually have access to. So because people didn't have access to it, they 
were very reluctant to share. Now that they know that they're going to be able to have 
access to it, they're more open in their sharing. 

One school district used QSP to investigate how well their local assessment 
could predict student performance on state tests. The results helped them revise 
their instructional planning and align local assessment with the state standards: 

We're relying on our local assessments, but we're also mindful of the impact of the state- 
mandated tests as well. So we're not only looking at the local assessments, but we're 
looking at how well the local assessments are predicting how well the kids will do on the 
PSSA and what we have to do.. ..we get more data, more real-time data that we can use to 
start to remediate right away and address those deficiencies. 

A school principal reported conducting "bi-weekly assessment with at-risk 
students to gauge progress over time." With the assessment information, this school 
offered "a before-school math tutoring program" and "sent out to the parents home 
support activities" to involve parents in students' learning. 

Other school decision makers began to envision new ways of paying close 
attention to students' needs and using student data to design new school 
improvement plans. One school had planned to "use QSP to accumulate student 
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data over the years beginning from next year" so that eventually, they could 
"conduct longitudinal analysis to examine student trends in learning." Another 
school was planning to use QSP to store and share student information across 
grades, so that "a fifth-grade teacher could have access to the test results from the 
fourth grade" and plan instruction based on each individual student's performance 
in previous years. 

Another school, in the early stages of using QSP, included administering 
periodic local assessments in their school improvement plan. They had set up goals 
to provide identification and early intervention of at-risk students by examining 
both student performance data from previous year and data from local assessment 
in the next year. 

It'll be at the beginning of next school year when we're once again hit with preparing for 
the year. It would be nice if a teacher could, over the summer, if they chose to go in and 
access their children, and start to look at things, and start to manipulate data.... Then we 
work off of those plans. I would like to be able to do periodic checks, and so would my 
teachers. 

Promoting Communication and Collaboration 

In the study, it emerged that QSP had acted as a catalyst for collaboration 
among school personnel. QSP afforded opportunities for inquiry and reflection 
among colleagues, which can have a great impact on improving achievement for all 
students (Hall & Hord, 2001). 

Several districts and schools had formed data use teams and participants 
worked together to make decisions about which data to collect or import, what 
questions to investigate, how to analyze data and present results, and what actions 
to take to improve instruction. 

For example, a school principal who participated in QSP training with two, 
fourth-grade classroom teachers, the reading specialist and the guidance counselor, 
described how after the training, she and her colleagues worked collaboratively to 
discuss how they planned to implement QSP based on their school improvement 
plan: 



We came back to school, and we did talk about how we could use the program based on 
our school improvement plan, and kind of the goals that we have for learning here, and 
how we would use that for instructional purposes. 
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In addition to collaboration among educators, there was evidence of the 
importance of collaboration between information technology personnel and the 
decision makers. A district director of instruction highlighted that to get to the point 
of being able to use the data to plan for students' learning required the support of 
the technical department to clean the database, import the data, and offer continued 
support to school principals and teachers: 

I would not have been able to do that if I were just on my own... it does take sort of a 
technologically astute mind that's familiar with databases and how they work, and how 
that system works. Fortunately, I have a man who could do it. Anything I've asked for 
he's been able to produce in a matter of moments, but if he were to leave. I'd pretty much 
be back to square one. 

A factor that schools recognized as important in meeting students' needs was 
collaboration between parents and the school. Some schools were planning to utilize 
web-based QSP to deliver more timely student progress reports to parents and to 
increase parental involvement in their students' learning. For example, one principal 
stated that: 

The web-based component is very important to me.... We have to do the progress 
reports. No matter when we put them out, there are always some parents who come back 
and say, "If we'd only known sooner, we could have helped and done something." 

Training and Professional Development 

For the initial national rollout of QSP, CRESST's training program formed the 
core of the training given to key personnel at the district or school level. CRESST's 
training program was also employed to train trainers. Trainers conducted face-to- 
face training for a considerable majority of survey and interview respondents. 

In rating the effectiveness of their training in preparing them to implement 
QSP, on a scale of 1-8 (with 1 as "not at all effective" and 8 as "extremely effective"), 
62% of survey respondents rated it as "effective" or "very effective," while 38% 
regarded the training as "not very effective" or "not at all effective." 

The interview data revealed that participating in the CRESST training program 
had changed some district and school personnel's perception of what constitute data 
and why data collection should be guided by research questions. For example, one 
principal pointed out that after participating in the training, he had a much better 
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understanding of what data were and how to manipulate data to answer his 
questions: 

It has changed the way that I've worked with data. I've actually gone from [being] an 
individual that didn't work with data to an individual that's now working with data. The 
understanding of what's going on with the data is a lot clearer now. 

In another example, a district administrator revealed that he had repeatedly 
advised colleagues in his school district not to input all kinds of data into the district 
database for analysis without giving serious consideration to the questions to be 
investigated. He observed that QSP training was effective in making people realize 
the role research questions play in the cycle of investigation, and as a result, a more 
systematic approach to inputting data had been adopted: 

And I think finally at the training they saw the light that these were the only assessments 
they needed in the system and not try to put everything under the sun... So that was, I 
think, a benefit of the training sessions. This will give them the understanding that it 
depends on the type of question and not just having all of the data and dumping it in. 

However, one theme clearly emerged from the interview data, perhaps best 
typified by a user who said, "It was helpful training, but I would have liked to 
receive training after implementation." Most users needed continued support to 
implement QSP successfully. Interview participants cited numerous examples where 
district employees needed to provide support to QSP users. In some cases, this 
involved having a district employee working with people at the school or building 
level after the training. For example, one district administrator stated that he makes 
suggestions for the types of analyses that can be conducted: 

We will typically tell them, "This is the kind of information you can get out of that," and 
we'll give them some suggestions, but we will also encourage them if we don't answer 
the questions that they have, to have them go ahead and experiment. 

In another district an administrator had scheduled school visits each month 
during the initial stages of implementation: 

And like anything else, when we go to visit the schools just like in the classroom, the 
schools that have been using it, pull it up, go to this and that, and then they ask us their 
question. "How come this? How could I do this?" Where other schools are fumbling 
around it's because they haven't used it. I think the site visits have been and I think the 
principals will tell you that the site visits have been by far the most valuable. 
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Whatever, the form of continued support for implementation, it required 
continuous communication between users and more expert users. As one district 
employee put it in relation to his work with schools: 

They're all working on trying to understand. They've been in constant communication 
with us. 

Despite the limited resources at CRESST for follow up training, CRESST 
personnel did manage to give most district administrators sufficient continued 
support by telephone and web communication to enable them to provide other 
district and school personnel the help they needed for implementation. 

However, more than one-third of participants said that they received support 
from outside sources, other than personnel at CRESST. Interviewees reported that 
they had sought help from outside the district (mainly from QSP trainers) for help 
preparing and importing data and for continued help with data analysis. A few 
participants indicated that they had used the online training as a "refresher" after 
the initial training, but the data from the survey and interviews point to not only the 
need to shore up technical skills associated with data preparation and import, but 
more importantly, the need for continued training and support in data analysis 
skills. These data are compatible with previously reported survey data (see Table 4), 
which indicated that a majority of respondents did not believe that their district had 
either the technical support, or the capacity, to use data effectively without outside 
support. 



Discussion 

The design of QSP as a decision support tool in education is a visionary one. It 
invites participants at all levels of educational decision making to be players in the 
input and analysis of data. Though it may be a long-term process involving 
significant investment in intellectual capital, QSP has the potential to encourage all 
participants to buy into the practice of data analysis and data-based decision 
making. 

The results of the evaluation described above represent a view of the very first 
uses to which QSP has been put by district and school administrators. They show 
first of all that QSP has been found useful and is being used by a significant 
proportion of the administrators who have been trained in its implementation. They 
also show that it is being used for a cluster of analyses that are closely related to 
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NCLB — disaggregation, identification of low-performing students, and prediction of 
scores on statewide tests. While respondents listed a variety of purposes for data 
use, few interviewees described activities that went beyond the identification of at- 
risk students. In contrast, the potential uses of QSP for examining the factors 
contributing to student achievement, and for formative assessment, were very much 
less utilized. We hope and expect that these more sophisticated uses of QSP will 
emerge as users' familiarity with the program increases, as additional data elements 
are entered into the system, and as training in QSP becomes more widespread at the 
classroom level. 

QSP was developed and supplied at no cost to end users. However, users 
encountered significant front-end costs in moving forward to implementation. Our 
users commented extensively about the personnel time and effort in preparing, 
cleaning, and importing data — indeed this proved an insurmountable hurdle for 
some. As numerous software start-ups have shown, it is often necessary for 
providers to support end-users for significant periods — particularly during periods 
when costs appear to exceed benefits— before users can be self-sustaining and the 
benefits from the tool become recognized and ultimately perceived as indispensable. 
Many of our surveys and interviews were conducted at the pump-priming stage of 
this process during which users had to be trained both in the mechanics and in the 
value of the output, and when high costs associated with data input were being 
experienced. 

Our users were clear that, while the CRESST training was effective, continued 
IT support and data analysis skills training would be important elements in 
continuing and expanding use of QSP. A significant minority of our users who had 
the resources had contracted with third-party providers for continued support and 
training. This underscores that considerable investment in human capital will be 
required to develop assessment literacy and data analysis skills that will, ideally, 
reach from district to classroom level. 

Our results suggest that web-based QSP represents the opening salvo in a long 
campaign toward evidence-based decision making in schools and districts. The use 
of QSP to service the objectives of NCLB is an important starting point for this 
process, and one around which additional skills in decision making will crystallize 
and grow. Creating the conditions for effective data use is a necessary precursor to 
the widespread reality of data use as an engine of school improvement. 
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Objective 5: Design Requirements for the 5 th Generation 

of Decision-Support Tools Using Results From the Foregoing Objectives. 

From our research evaluation of QSP implementation, from CRESST and other 
research, and from the project team's ideas that arose during the development of 
QSP, we have identified a number of features and functions to include in a 5 th 
generation version of QSP that would considerably expand its utility. These are 
presented in two sections: a) proposed changes and enhancements to the existing 
features and functionality of QSP, and b) new features to add to QSP. 



Proposed Changes to Existing Features 

Student-centric. The original conception of QSP was as a decision support tool 
for use at the school and district level. Therefore, the database in the current version 
uses a school-centric organization. This means that each student in the system is 
"locked" to a particular school in the state/ district/ school hierarchy, and that the 
school is the smallest unit of analysis (see Figure 2). 



District 1 



School 1A School IB 



Student 1A.1 




Student 1B.1 


Student 1A.2 




Student IB. 2 


Student 1A.3 




Student IB. 3 


Student 1A.4 




Student IB. 4 



District 2 



School 2A School 2B 



Student 2A.1 




Student 2B.1 


Student 2A.2 




Student 2B.2 


Student 2A.3 




Student 2B.3 


Student 2A.4 




Student 2B.4 



Figure 2. School-centric organization of QSP. 

We would propose to change this structure in a 5 th -generation version of QSP 
to a student-centric organization. This means that each student in the system would 
only be associated with a permanent ID number and would be outside the 
state/ district/ school hierarchy (see Figure 3). 
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District 1 



District 2 



Student 001 
Student 002 
Student 003 
Student 004 
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School 2B 



Figure 3. Student-centric organization. 

The main benefits of this organization would be that duplication of students in 
schools or districts would be eliminated, and transferring and promoting students 
between and within schools or districts would be much easier. Currently, if there are 
any differences in the student ID, QSP will consider the difference as additional 
students, and system administrators must reconcile student IDs when transferring 
students into their system from another district using QSP. For large districts, this 
can be a time-consuming process. Additionally, making students active or inactive 
in the database would be an easier process in a student-centric system. Also, it 
would be simpler for administrators to back up and restore data that was organized 
in a student-centric way. 

There would be advantages to using the gradebook from a student-centric 
structure. In the school-centric organization of QSP, students stay within a teacher's 
gradebook even after completing a class. Making QSP student-centric would enable 
system administrators to transfer a student's classroom-level data from the 
gradebook from last year's teacher(s) to next year's teacher(s). 

Currently, QSP stores all portfolio items in one folder. In a student-centric 
version of QSP, each student would have a unique folder to save their portfolio 
items, which would enable students to access an individual archive of their portfolio 
items, and to transfer the portfolios when they moved to a new school or district. 

Hybrid - Avalon 

QSP was written using the latest available technology, including the .NET 
framework and Visual Studio .NET application-development environment. All the 
code resides on a server, and all processing is done on the server. Because of this, the 
software can slow when large numbers of users access an individual server 
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simultaneously, and when the server has to process a report for more than 50,000 
students. A 5 th -generation version would provide a solution to this by employing 
hybrid software, where the software is stored both on a desktop and on an Internet 
server. This would improve the speed of the software by offloading part of the work 
to the desktop when the server is busy. This would also make the process of 
upgrading QSP easier because the desktop portion of the software could 
automatically check for updates to the server and notify the user. One caveat is that 
a hybrid version would only work on PCs. However, Macs running Virtual PC 
could run this hybrid version in emulation. 

Database Management Access (DBMS Access) 

The QSP software includes both a database and an interface to manage and 
report data. When implementing the current version, system administrators are 
required to import all of their data into the QSP database. This database is stored on 
a Microsoft SQL server. 

During the implementation of the current version of QSP, many system 
administrators noted that they wanted more options for storing their data. Most 
schools and districts already maintain a database and, because of time constraints, 
do not want to export data out of their database and import them into QSP. While 
many QSP sites are using SQL, a significant number are using other products that 
are incompatible with QSP, including Oracle and InfoMix. To resolve these issues, a 
5 th generation of QSP would include DBMS Access. This would enable system 
administrators to use the QSP interface to manage and analyze data in their existing 
database system. 

Importing Data 

With the current version of the system, QSP administrators are able to import 
student, teacher, and gradebook data using an In/ Out tab. This tab contains wizards 
to take administrators through the step-by-step process of conducting an import or 
export of data. 

Many users of the QSP software have expressed that they would like the data 
import process to be speeded up. To accomplish this, a 5 th generation version of QSP 
would include an alpha-numeric command structure. A command structure is a set 
of programming commands that enable a computer to communicate with a server. 
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With the inclusion of alpha-numeric characters (e.g., letters and numbers), large data 
sets would be partitioned into smaller files to improve the accuracy and speed of 
importing data, and administrators would be able to import data from other 
software, such as Excel and SPSS. 

Within the current version of QSP, data is imported using a top-down method. 
In other words, you can only view data at the level imported or lower (e.g., district 
data can be viewed at the district or school level). By updating the QSP database to 
use a bottom-up method, student- and school-level data would be accessible at the 
district level. 

Default Variables 

All data in QSP are stored according to variables. These variables describe a 
characteristic that a student may have. For example, the variable "LEP" tells you 
whether a student is Limited English Proficient or Fluent English Proficient. 

Within the original desktop version of QSP, system administrators were 
limited in the number of variables that they were allowed to create. This included 
some default variables for use with all users of QSP. Once system administrators 
used their allotment of variables, they were required to delete old variables before 
creating new ones. In the web version, to allow for a larger number of custom 
variables, the number of default variables was reduced so that system 
administrators could create new variables whenever needed. A 5th generation of 
QSP would increase the number of default and non-default variables. This would 
permit default variables to be easily mapped to School's Interoperability Framework 
(SIF) variables as the SIF specifications change. With default variables mapped to SIF 
variables, users would be able to enter data once and propagate data between QSP 
and other software. 

To allow for maximum flexibility in querying and analyzing data, in QSP 
version 4.0, users have to identify the elements of the report before it can be 
generated. Adding more default variables would enable the next generation of QSP 
to include pre-built reports. Many users have requested this feature, especially for 
NCLB reporting and for more novice users of the system. Reports that are required 
for NCLB reporting could be pre-built in the system. Other pre-built reports could 
include a series of existing reports that answer specific, pre-determined questions. 
To answer the question, "What is the achievement of our students in math?" QSP 
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would generate a series of reports. These would include pie, floating bar, score 
range, etc., reports that used default or custom variables from the system (e.g., math 
norm-referenced test, the state math assessment, the district math assessment), 
broken out by certain subgroups. 

Accessibility 

QSP 4.0 includes 10 types of school level accounts. These provide differing 
levels of access for QSP state, district, and local users, who include administrators, 
principals, program coordinators, teachers, counselors, office staff, students, and 
parents. Different functions are provided using tabs for each of these school-level 
accounts. For example, teacher-level users have exclusive access to the Gradebook 
tab. 

During the implementation of QSP 4.0, it became clear that the current system 
of accounts needs to be modified. The current system assumes that all users at a 
certain level will have the same skills and needs when using QSP to analyze student 
data. The users of QSP vary considerably in their abilities involving technology and 
data analysis. In the original QSP design we had planned to customize access to 
QSP. For example, a principal account could have access to the In/Out tab in 
addition to their tabs designed for principal use. However, this was not technically 
feasible during the development of version 4.0. In the current version, users are 
assigned a primary role and the possibility of two sub-roles. When the user logs in to 
the system they have a choice of roles, but if they want to change roles, they have to 
log out and log back into the system. A 5 th generation would permit users to 
combine all the roles into one. 

Report Options 

With the current system, users are required to manually select the format for 
each report. As the research team found during interviews with QSP users, this can 
be a difficult process for those who are less experienced with analysis. Not only do 
they need to be able to follow the step-by-step process in creating the report, but also 
they need to be able to match the report back to their research questions. 

A 5th generation of the Reports tab would scaffold users in determining which 
report best fits their analysis. Users would be given the option of utilizing a report 
wizard, which would ask users about their research question, groups, variables, and 
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time periods. Once the questions were answered, it would provide a list of 
appropriate reports. 



An additional report would be included in a 5 th generation— the explore report. 
This would be an interactive report that lets users query the database about a 
particular group and use the explorer (a decision tree), to investigate indicators 
about the group. The explorer would allow a user to disaggregate according to a 
categorical variable. For example, users could go to a list view (see Figure 4) where 
they select a variable from the pre-determined common indicators and could see a 
pie chart of the distribution. 
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Figure 4. Explore report. 



Flexibility for School Type 

The online version of QSP is currently designed to organize data according to 
traditional school types. Whenever a QSP administrator creates a new school, they 
must select elementary school, middle school, or high school. 

Some schools in our study did not fit into traditional grade-level categories. 
This supports findings by the National Center for Education Statistics (2002) that 
shows a shift in the structure of public schools in the United States. For example, 
during the 2000-2001 school year, 18% of all elementary schools included grades 
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traditionally considered both elementary and middle school. Furthermore, 6% of all 
public schools were classified as combined elementary and secondary schools. 

Increased flexibility for school type would be included in a 5 th generation of 
QSP. Modification would be made to the School tab to enable system administrators 
to assign alternative school types to elementary school (K-6), middle school (7-8), 
and high school (9-12). To accomplish this, a wizard will be provided in which 
administrators can set the grade levels to be included in an individual school. 

Standards Tab 

Using this tab, system administrators can define grading standards for subjects 
and domains. Teachers can show progress toward state, district, or school standards. 
In a 5th generation of QSP, this function would be adapted so that standards can be 
tied directly to a cohort of students for a given school year. This would provide the 
benefit of accounting for changes in standards as students' progress through school, 
so that student performance can be measured against the standards that were in 
place when they entered the school as well as any standards that were subsequently 
introduced. This was a feature that districts in several states requested. Changes 
would also need to be made to the Gradebook tab so that teachers could monitor 
progress against the range of standards. 

Additional Features and Possibilities 

Value-added analysis. CRESST researchers have been developing value-added 
models that could be incorporated into QSP. Using a value-added system, the 
emphasis is switched from measuring current proficiency to measuring gains. In 
value-added gains or growth, the status at a certain point of time (i.e., initial status) 
can be a strong and important factor. For example. Figure 5 shows the growth 
trajectories of students with differential starting levels at a given time point (t) and 
with varying gains at a later time point (t+1). 
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Figure 5. Distribution of student achievement (Choi, Seltzer, Herman, & Yamashiro, 2004). 



The first graph in the figure shows the achievement gap closing — from a high 
initial status the gains were low, whereas from a low initial status the gains were 
high. The second graph shows medium gains from both the high and low initial 
status. The third graph shows high gains from high initial status and a low gain 
from a low initial status — a widening of the achievement gap. These growth 
trajectories would not be shown if only measuring the current proficiency levels of 
students. The CRESST value-added model enables schools and districts to monitor 
growth trajectories for each school. Figure 6 shows the individual growth trajectories 
in math for 16 schools in a district. 




Figure 6. Growth trajectories in math for 16 schools. (Choi et al., 2004). 
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A value-added analysis capability in QSP could answer questions such as: 



• Are students with high initial status growing faster than students with low 
initial status? 

• Are schools with high initial status growing faster than schools with low 
initial status? 

• Are there any schools that start low initially but grow faster compared to 
others? 

A 5 th generation of QSP would include value-added analysis capabilities and 
specific report formats to display the initial status as well as the gains in student 
achievement from the initial status. 

Financial Data Analysis 

Data through which program expenditures have been evaluated are most often 
aggregated at the district level or above and typically averaged over a range of 
students and programs. For more than 10 years, researchers have argued that data 
disaggregated at the student level are more useful for evaluating resource 
effectiveness than data averaged at the district level (e.g., Berne & Stiefel, 1995; King 
& MacPhail-Wilcox, 1994; Monk, 1992), and that longitudinal evaluation of student 
progress is a more sensitive means of addressing product function questions than 
traditional annualized measures. A 5 th generation of QSP would give educators the 
capacity to use longitudinal financial data linked to individual students to analyze 
the impact of resource allocation on student achievement. 

Fiscal variables would be included and QSP would perform multiple 
regression analyses to determine a causal relationship between critical financial 
variables and student outcomes. To accomplish this, we would add an expenditures 
wizard that enables users to easily allocate financial resources, a tool for statistical 
analysis of financial data, and new reports to display the data. The expenditure 
wizard would have three components: selection of a students' group, selection of 
fiscal variables, and an entry for financial cost (see Figure 7). 
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Figure 7. Expenditures wizard. 



Integrating QSP and the Education Data Exchange Network (EDEN) 

While first and foremost a tool for supporting school improvement, QSP has 
the potential to be integrated with EDEN and become an essential part of the 
education information system. QSP could serve as the baseline data engine feeding 
school and classroom level data into the information pipeline up to the district, state, 
and federal levels. Integrating QSP with EDEN would provide education program 
evaluators at all levels of the system with a network to capture, aggregate, and 
upload student-, classroom-, and school-level data and make them useful across all 
districts and states in the nation. 

A 5th generation of QSP would include an EDEN tab and EDEN elements 
would be included as system variables. Specific functionality would be added so 
that reports on common data elements required at the school, district, state, and 
federal level could be generated. At each information transfer point in the system, 
the data from the prior level would be aggregated and mapped to the relevant 
standards. Existing QSP reports could be used with the level of aggregation 
highlighted (see Figure 8). 
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Figure 8. Model of data aggregation through the information system. 



Assessment Builder 

To improve teachers' assessment capabilities and the quality of classroom 
assessments, CRESST has developed an online authoring system that can be used to 
design complex performance assessments across subject areas and grade levels. The 
system enables teachers to access a database of reusable assessment objects that 
"know" what they are useful for and the purposes and contexts for which they are 
most likely to be valid. The assessment objects contained in the system database 
include information sources such as text descriptions, graphic illustrations, and 
animations of physical phenomena, and interactive simulations permitting 
investigation of phenomena. Teachers can combine these information sources with a 
variety of prompts and questions to create assessments covering a wide range of 
domain independent knowledge types and skills; e.g., explanation, recall, and 
application of conceptual and factual knowledge; problem solving; understanding of 
relations among concepts, facts, and skills; and transfer of knowledge to novel 
situations and problems. Assessments can also be structured to capture domain 
specific knowledge such as facts or concepts particular to the domain (see Figure 9). 
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Figure 9. Interface for assessment builder. 

In a 5 th -generation tool the assessment builder could be incorporated into QSP. 
Classroom-based assessment data could be stored in the database and queried, 
analyzed, and reported on along with other data types in the system. 

Lesson Planner 

In a 5 th generation of QSP there is the potential to add a planning tool for both 
teachers and students. Figure 10 shows the structure of the planning tool and its 
relationship to other elements in QSP. The planner would provide teachers with 
easy access to a number of resources for instructional planning: content and 
performance standards, goal setting capacity, the assessment builder, an archive of 
opportunity to learn (OTL) and lesson plan information, and an interactive 
community board. For students the planner enables students to engage in 
monitoring and evaluating their own learning. This practice is increasingly 
advocated as a means to developing metacognitive skills — skills students use to 
better understand their own thinking and learning processes (Black & Wiliam, 2004; 
NRC 2001; Sadler, 2001). An additional benefit of the planner is that teachers and 
student would have increased communication opportunities to frequently give and 
access feedback outside the instructional time of the classroom (see Figure 10). 
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QSP Planner 



Standards Tab 



Standards Tab 



Assessment Builder 1-^- 



OTL Library 



C 



Lesson Plan Library 



) 



^ Community Board 



Goals Tab 



1. Select Standards o- 



Content and/or performance standards 
State or district standards 



2. Select Benchmarks 

Choose from a preset list 
Create your own 



3. Set Goals 

Each student and/or class 



4. Select Formative Assessments 

Choose from a preset list 
Create your own 



5. Determine OTL Factors 

Choose from a preset list 
Create your own 



6. Plan Instruction o- 



Create lesson plan and strategies 
Select resources 



7. Implement Instruction 

Administer assessments 



8. Monitor Goals 

Not achieved (re-teach, reassess) 
Achieved (move on) 



-► Student Planner 



-► ( Student Planner 



► Student Planner 



^ ► ( Student Planner 



^ ► I Student Planner 



teacher sends data ► receives data ► both ► 



Figure 10. The teacher and student planning tool. 



From the Standards tab in the gradebook, teachers would first identify the 
standards that the period of instruction is going to address and then determine 
benchmarks toward those standards. This information would be communicated to 
each student through the student planner, along with the goals that the teacher has 
created via the Goals tab for each student and/ or the class as a whole. The students 
would be able to give feedback to the teacher about the goals and add some goals of 
their own. 

Each student would be able to upload work drafts, homework, reflections, and 
questions for the teacher to the digital portfolio and through the student planner 
would be able to give evaluations of their work and ask questions of the teacher. 
Teachers could provide feedback to suggest improvements, to guide or redirect the 
students' thinking, and to correct misconceptions. Therefore, students would not 
always need to meet directly with teachers and could receive feedback in different 
locations at different times. 

With the assessment builder, teachers could decide which assessments they 
will use to give them formative feedback that will guide their instruction to meet the 
standards. When the teachers have administered and interpreted the formative 
assessments, teachers would give individual feedback to each student via the 
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student planner, and students would be able to monitor and evaluate their own 
progress and goals and communicate their evaluation and plans to the teacher. To 
plan their instruction to meet the standards and goals, teachers may want to draw 
from the opportunity to learn (OTL) library, an archive of programs and strategies 
accumulated by them and their colleagues, as well as a similar archive of lessons. 
They may also use the community board, an interactive forum to ask questions and 
receive suggestions and advice from their colleagues about their instructional plans. 
During the period of instruction, teachers would administer formative assessments 
to determine student progress toward their objectives and to adjust curriculum and 
pedagogy if needed. 

At the end of the instructional sequence, teachers and students would 
determine if the goals have been met and decide if further instruction is needed or if 
it is appropriate to move on to another set of standards and objectives. The planner 
feature would make QSP a comprehensive tool for instructional planning. 

Speech Recognition Technology 

Adult speech recognition technology is widely used in the commercial sector. 
Currently, with NSF funding, CRESST is collaborating with the computer science 
and electrical engineering departments to develop child speech recognition 
technology for use with literacy assessments. 

A 5th generation of QSP could utilize both adult and child speech recognition 
technology. For example, rather than entering data manually, teachers could record 
student scores and their observations of student performance in the classroom. 
Students could directly record audio data into their portfolios. This could benefit all 
students, but particularly younger students who may not yet have the technology 
skills to include their data in the system. 

Conclusion 

QSP is a visionary tool that affords educators the opportunity to use data 
effectively for school improvement. From the outset of the project our intention, 
although ambitious, was to examine the impact of QSP on student learning. 
However, we were prevented from doing this by a number of factors. First, the time 
taken by districts to implement QSP was more extensive than originally anticipated 
because of the factors described earlier in this report. Second, school districts had an 
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urgent need to allow all willing schools to participate, which meant that there were 
considerable variations in schools' readiness to utilize QSP. Third, the instability of 
tests in many of our districts made comparisons difficult. Nonetheless, the study of 
the impact of QSP on student learning remains a future goal. 

Although we were not able to examine the effect of QSP use on student 
learning, our implementation study showed that most participants had found QSP 
useful in performing data analysis and that many of them could see the potential of 
the tool beyond their current practices. Indeed, for the most part, those participants 
who were no longer using QSP thought that QSP's features and functions were 
useful. Our study also revealed that NCLB had provided the impetus for data 
analysis for most of our users, and with only a small number of exceptions, their 
analysis consisted of descriptive statistics of the performance of sub-groups of 
students. This may prove to be a useful starting point for focusing on data as a 
vehicle to improve student achievement, and may lead users to more sophisticated 
uses of data in the long run, but at present, we have concluded that the full 
complement of the features and functions of QSP is not being used. 

Consider this scenario: At an end-of-the-year meeting, a group of fourth-grade 
teachers use QSP to begin an investigation that will help them make plans for their 
incoming class next school year. They begin by disaggregating the statewide reading 
scores by the subscales of the test. The teachers discuss the profile of strengths and 
weaknesses they see in the data and decide to see if the same strengths and 
weaknesses show up on the district-wide reading inventory. They choose a line 
graph to display the quarterly scores, by subscales, of the district reading inventory 
that the third-grade teachers conducted with all their students. Next, they use the 
scatterplot report to examine the relationship between the state and district reading 
scores and find that they are positively correlated. 

However, they want more specific information. They use the score range report 
to show the ranges for the district-wide test. Beginning with the students who 
scored less than 30, they identify the specific students who comprise this group. 
They open up the digital portfolio and watch video of individual children reading a 
graded passage aloud, followed by the child's retelling of the narrative. They do this 
for several students, and they begin to see some patterns in performance, especially 
in the area of fluency and in the ability of the students to recall the narrative 
sequentially. For some of the students of particular concern, the teachers want to 
examine the rate of progress and go to the students' individual history to see the 
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scores from the state and local assessments in reading for the past 3 years. Some 
teachers decide to access the standards-based student profiles and progress reports 
from the third-grade teachers to see how students' performance was rated in third 
grade. 

With the accumulated information, they begin to make plans for the incoming 
students. Using the Goals tab they identify specific goals for the year, establish the 
targets to achieve the goals, how often the targets will be assessed and how, and 
decide on the specific strategies they will employ to realize the targets. They will use 
the goals report to monitor progress toward meeting the goal. 

The current uses of QSP that were identified in Section 4 of this report are at 
some considerable distance from the above scenario. While QSP users may 
appreciate the potential of the tool to perform the kind of analysis described in the 
scenario, essentially, the capacity of QSP is beyond the current capacity of most of its 
users. 

Modifications to the tool to "scaffold" users have been identified in Section 5. 
Primarily, these would involve pre-determined questions and pre-built reports and 
would help those who are inexperienced in data use to get started. Nevertheless, 
longer-term considerations remain. Although most of our users thought the training 
they received was useful, it did not make up the shortfall between their data skills 
and the capacity of QSP. To achieve a better match between the skills of users and 
the capacity of the tool will require much more sustained support for training at all 
levels of the profession. In our study, we saw some promising examples of support 
beyond the initial training from both consultants and district personnel. 

There are clearly areas where QSP's functionality can be improved, especially 
the speed of importing the data, the compatibility with other databases, and the 
speed of the software when the server is accessed by a large number of users 
simultaneously. All of these are described in Section 5. Additionally, there are a 
number of features that would expand the capacity of QSP. These features include 
value-added analysis and financial data analysis capabilities. Yet, before the benefits 
of a 5 th generation can be fully utilized to improve student performance, the skills of 
the education professionals must be shored up through increased training and 
support in data use at all levels of the profession. Without this, the capacity of QSP 
will remain underutilized, and the potential benefits of the tool for students will not 
fully accrue. 
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APPENDIX A 



HISTORY SUBTAB OF THE STUDENTS TAB 



QSP1 



Asimov Elementary 

Hilda Chavez / Teacher / 14 -IS 



help 

logout 



Home 


f 

Reports 


Goals 


Groups 


\ 

Gradebook 


Students 










Class : All Classes (02-03) v Group 


: All Class v ■ 









Students 


History 


Portfolio 


Parents 



View the past 2 years v print edit 



Name A | 1-12 v 


Ackart, Karen 


Barton, Larry 


Weiss , Michael 


Jones , Weston 


Ackerman, Chris... 


Keystone, Devon... 


Fahidquar , Moham... 


Gonzales , Raul 


Lopez , Emmanue... 


Juarez, Felipe 


Deener, Matthew 


Ackers, Terence 


20 active students 



Variable 1-10 v 


13-14 


14-15 


LEP Test 


36 


52 


Math Efficacy 


Low 


Medium 


SAMPLE Test Date 1 


11/13/2001 


11/12/2002 


SAMPLE Test Date 2 


01/17/2002 


01/16/2003 


School Safety 


Medium 


Medium 


Reading Attitude 


Missing 


Missing 


SAMPLE Test Date 3 


02/14/2002 


02/13/2003 


State Reading Test 


94 


153 


State Math Test 


82 


149 


State Language Test 


72 


154 


16 Variables 
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APPENDIX B 



DIGITAL PORTFOLIO 
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APPENDIX C 



GROUPS TAB 



QSP* 




1st Beta Elementary School 




help 

logout 






Teacher ; Zack / Semi , 03-04 




Home 


Reports [ Goals 


Groups 


Gradebook 


Students 







Data About 



Parent 



0 



Ethnicity 


Name 


Type 


A Ethnicity: Invalid 


System 


A Ethnicity: Valid 


System 


41 Ethnicity: Missing 


System 


A Ethnicity: N.A 


System 


A White 


System 


A Black 


System 


A Hispanic 


System 


A Asian 


System 


A Pacific Islander 


System 


A American Indian 


System 


A Other 


System 
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APPENDIX D 



SAMPLE REPORTS FROM 24 
REPORTING OPTIONS 



Reports 




23 different graphic reporting options 
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APPENDIX E 



GOALS TAB 



QSP* 


Georqe Washington High School 


help 


t 


Clayton R. Skinner / Semi, 02-03 


logout 



Goals 



t 



Edit Goal 



cancel 



Goal Properties 



edit 



Name 

Monitor 

Start 

End 

Main Group 



Language Arts Goal 
Karen Stone 
Semi, 01-02 
Sem2, 05-06 
All School 



For the Goal to be achieved, F3] 
of the Targets must be achieved. 

(1 mandatory, 3 optional) 



Targets 



Name 



Target 1 



Asian Reading Ta 



Caucasian Sessii 



Reading Target 



edit 
delete 
view + 
view - 



Priority 



normal 



mandatory 





normal 




normal 
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APPENDIX F 



STANDARDS SUBTAB OF THE GRADEBOOK 



QSF 




George Washington High School 


help 

logout 






David Jones /Semi, 01-02 


r-- i 

Home 


Gradebook 


Students "| Groups | Goals j Reports 


' * 


Class: | Per2 English (Semi, 01-02) 


jJ Group: | All Students jJ 


preferences 





Variables 


Standards 


Students add 




Name 1 1 - 12 


1.2 


1.3 


1.4 


2.0 A 


2.2 


2.3 


2.4 


Jones, Seth 


8.90 


5.55 


8.88 


5.55 


5.77 


8.67 


7.34 


Jones, Stuart 


7.75 


8.23 


7.45 


6.35 


6.55 


9.66 


6.89 


Jones, Guadalupe 


6.35 


8.06 


7.33 


6.55 


7.34 


4.89 


6.50 


Jones, Stan 


7.50 


6.55 


6.89 


6.89 


7.22 


8.11 


9.02 


Jones, Owen 


6.00 


7.34 


6.50 


7.05 


8.56 


7.25 


9.28 


Dunks, Ashley 


8.67 


7.22 


9.02 


7.50 


6.35 


7.33 


9.66 


Jones, Aba 


9.66 


8.56 


9.28 


7.91 


7.50 


6.89 


4.89 


Jones, Simon 


4.89 


5.62 


5.77 


8.06 


6.00 


6.50 


9.66 


Jones, Fred 


7.90 


6.10 


6.96 


8.11 


9.02 


9.02 


4.89 


-- Jones, Aaron 


5.08 


7.24 


6.44 


8.23 


7.24 


5.08 


6.44 


Jones, Jim 


8.99 


8.11 


8.65 


8.88 


5.77 


6.55 


9.28 


Jones, Sue 


7.05 


7.25 


7.91 


8.99 


6.96 


7.34 


5.77 


29 active students 


< ► 
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